Information processing apparatus and information processing method

ABSTRACT

An information processing apparatus comprises: an acquisition unit configured to acquire a viewpoint position of an observer who observes a first three-dimensional virtual object arranged in a virtual space and having at least one cross section; an acquisition unit configured to acquire a normal vector of a first cross section of the first three-dimensional virtual object; an image generation unit configured to generate, based on the viewpoint position of the observer and the normal vector of the first cross section, a second three-dimensional virtual object having a second cross section with a different normal vector from the first cross section; and an output unit configured to output an image of the second three-dimensional virtual object generated by the image generation unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a technique of supporting an operationto a three-dimensional virtual object arranged in a virtual space.

Description of the Related Art

Conventionally, three-dimensional CAD systems are used in the field ofmanufacturing and design, and can design a component and its crosssection while three-dimensionally displaying them. For example, there isknown a technique of, when observing a cross section, setting theposition and direction of a cross-sectional plane serving as a reference(to be referred to as a cross-sectional operation hereinafter), changingthe position of the cross section of a component by an operation on aCAD screen, and observing the cross section.

A virtual reality (VR) system has also been studied, in which design isperformed by using a three-dimensional CAD system in a virtual realityspace by using a head-mounted display (HMD) (Japanese Patent Laid-OpenNo. 2012-53631). Further, there is proposed the use of a mixed reality(MR) system in which a physical space and a virtual space are combinedby using an HMD (Japanese Patent Laid-Open No. 2010-66898). An operationdevice that is operated by gripping it with the hand or the like, like agame controller, instead of a mouse or a keyboard, is also used in anenvironment where an observer wears an HMD and observes a virtual objectwhile walking around a physical space.

However, in the cross-sectional operation during display in VR or MR, across section is operated with predetermined buttons of a gamecontroller. When the line-of-sight direction of an observer differs froma predetermined direction, the operation direction of the cross sectionbecomes opposite to the direction in which the observer views the crosssection, and the observer may not operate the cross section intuitively.

For example, a case in which a cross section is operated in a coordinatesystem defined by the origin and three axes (X-, Y-, and Z-axes) of acoordinate system defining a space will be considered. Assume that thegame controller is configured so that its up button is associated with amovement in the +Y direction, its down button is associated with amovement in the −Y direction, and a cross-sectional plane parallel tothe X-Z plane is operated in the Y-axis direction. In this case, whenthe observer stands on the X-Y plane and turns his eyes in the +Ydirection, the cross-sectional plane moves deep (i.e. away from theobserver) with the up button indicating the +Y direction and forward(i.e. toward the observer) with the down button indicating the −Ydirection. In this case, the operation direction of the observer and themovement direction coincide with each other, so the observer can performan operation naturally. To the contrary, when the observer turns hiseyes in the −Y direction (e.g. he turns around by 180°), thecross-sectional plane moves toward the observer with the up buttonindicating the +Y direction and deeper (away from the observer) with thedown button indicating the −Y direction. That is, the movement directionbecomes opposite to the operation direction of the observer. This isunnatural to the observer, and he cannot operate the cross sectionintuitively. In some cases, when the cross-section normal direction isnot toward the observer (i.e. in the plane normal to the direction ofthe line between the observer and the cross section), he cannot confirmthe cross section.

SUMMARY OF THE INVENTION

According to one aspect of the present invention, aninformation-processing apparatus comprises: an acquisition unitconfigured to acquire a viewpoint position of an observer who observes afirst three-dimensional virtual object arranged in a virtual space andhaving at least one cross section; an acquisition unit configured toacquire a normal vector of a first cross section of the firstthree-dimensional virtual object; an image generation unit configured togenerate, based on the viewpoint position of the observer and the normalvector of the first cross section, a second three-dimensional virtualobject having a second cross section with a different normal vector fromthe first cross section; and an output unit configured to output animage of the second three-dimensional virtual object generated by theimage generation unit.

The present invention provides a technique intending to provide anatural cross-sectional operation with respect to a three-dimensionalvirtual object arranged in a virtual space.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention and,together with the description, serve to explain the principles of theinvention.

FIG. 1 is a block diagram showing the overall arrangement of a systemaccording to the first embodiment;

FIG. 2 is a view showing the outer appearance of the system according tothe first embodiment;

FIG. 3 is a view for explaining the relationship between the viewpointand the cutting direction (to be referred to as a cross-section normaldirection hereinafter) of the cross section of a virtual object;

FIG. 4 illustrates an operation in step S8030;

FIG. 5 is a view for explaining an operation with respect to an upwardcross section in step S8080;

FIG. 6 is a view for explaining an operation with respect to a downwardcross section in step S8080;

FIG. 7 is a view for explaining an operation with respect to a lateralcross section in step S8080;

FIG. 8 is a flowchart showing processing to be performed by aninformation-processing apparatus 1040 according to the first embodiment;

FIG. 9 is a view of an operation device of another form;

FIG. 10 is a block diagram showing the overall arrangement of a systemaccording to the second embodiment;

FIG. 11 is a view of the outer appearance of a head-mounted display(HMD) according to the second embodiment;

FIG. 12 is a flowchart showing processing to be performed by aninformation-processing apparatus 10040 according to the secondembodiment;

FIG. 13 is a block diagram showing the overall arrangement of a systemaccording to the third embodiment;

FIG. 14 is a view showing the outer appearance of the system accordingto the third embodiment;

FIG. 15 is a flowchart showing processing to be performed by aninformation processing apparatus 13040 according to the thirdembodiment;

FIG. 16 is a block diagram showing the overall arrangement of a systemaccording to the fourth embodiment;

FIG. 17 is a view showing the outer appearance of the system accordingto the fourth embodiment;

FIG. 18 is a flowchart showing processing to be performed by aninformation processing apparatus 16040 according to the fourthembodiment; and

FIG. 19 is a view for explaining an operation in step S18040.

DESCRIPTION OF THE EMBODIMENTS

Preferred embodiments of the present invention will now be described indetail with reference to the accompanying drawings. Note that thefollowing embodiments are merely examples, and are not intended to limitthe scope of the present invention.

First Embodiment

An information processing system including a head-mounted display (HMD)will be exemplified and described as the first embodiment of a displaycontrol apparatus according to the present invention.

<Apparatus Arrangement>

FIG. 1 is a block diagram showing the overall arrangement of the systemaccording to the first embodiment. FIG. 2 is a view showing the outerappearance of the information processing system according to the firstembodiment. As described above, the information processing system uses ahead-mounted display (HMD) including a position and orientationmeasurement unit and a display unit. Note that a magnetic fieldgenerator 2050 that generates a magnetic field is arranged at apredetermined position in a physical space. A sensor controller 2070controls the operation of the magnetic field generator 2050.

A head-mounted display (HMD) 1010 includes a head position andorientation measurement unit 1020 serving as a magnetic sensor, and onedisplay unit 1030. The head position and orientation measurement unit1020 measures a change of the magnetic field corresponding to theposition and orientation (position and direction) of the head positionand orientation measurement unit 1020 itself. Then, the head positionand orientation measurement unit 1020 sends the measurement result tothe sensor controller 2070. The sensor controller 2070 generates asignal value representing the position and orientation of the headposition and orientation measurement unit 1020 in a sensor coordinatesystem 2060 based on the input measurement result, and sends the signalvalue to an information processing apparatus 1040. The sensor coordinatesystem 2060 (predetermined coordinate system) is a coordinate systemhaving the position of the magnetic field generator 2050 as an origin,and three axes perpendicular to each other at this origin as the X-, Y-,and Z-axes.

A viewpoint position and orientation calculation unit 1050 of theinformation processing apparatus 1040 calculates the position andorientation of a viewpoint in a virtual space coordinate system 2080 ofthe image of a virtual space that is displayed on the display unit 1030.Then, the viewpoint position and orientation calculation unit 1050updates a virtual information management unit 1060. The virtual spacecoordinate system 2080 is a coordinate system having, as an origin andX-, and Y-, and Z-axes, the origin and three axes of a coordinate systemdefining a virtual space where a three-dimensional virtual object isarranged.

Note that a measurement method using a magnetic sensor is employed as aposition and orientation measurement method. However, the presentinvention does not depend on the position and orientation measurementmethod, so a position and orientation measurement method using a sensorsuch as an optical sensor or an ultrasonic sensor, other than themagnetic sensor, may be adopted. A method of attaching an imagecapturing device to the HMD 1010 and estimating the position andorientation of the image capturing device from an image captured by theimage capturing device may also be employed. A plurality of position andorientation measurement methods may also be used in combination.

The virtual information management unit 1060 holds the position andorientation of the viewpoint calculated by the viewpoint position andorientation calculation unit 1050. Further, the virtual informationmanagement unit 1060 reads out pieces of information of a virtual object2090 and cross-sectional plane 2040 from an information storage unit1065. Pieces of position information of the virtual object 2090 andcross-sectional plane 2040 are coordinate values in the virtual spacecoordinate system 2080.

An observer (operator) who wears the HMD 1010 on his head controls thepositions of the virtual object 2090 and cross-sectional plane 2040 inthe virtual space via an operation unit 1160 (for example, a gamecontroller) serving as an indicator that is gripped with the hand or thelike and operated. The operation unit 1160 includes a direction inputunit 1150 configured as a four-way selector key capable of indicating atleast two axial directions (up-and-down and left-and-right in thiscase). The operation unit 1160 determines an input to the directioninput unit 1150, and sends the determination result to the informationprocessing apparatus 1040.

An operation direction acquisition unit 1090 acquires the inputdirection sent by the direction input unit 1150. A cross-sectionalposition direction acquisition unit 1070 acquires the cross-sectionalplane 2040 from the virtual information management unit 1060, andcalculates a normal 2100 to the cross-sectional plane 2040.

FIG. 3 is a view for explaining the relationship between the viewpointand the normal direction (to be referred to as a cross-section normaldirection hereinafter because it is a direction along the normal betweenthe observer and the cross-section in question) of the cross section ofthe virtual object. In a state 300 a, a cross-section normal direction3020 is not turned to an observer's viewpoint 3025 in the virtual space.In a state 300 b, a cross-section normal direction 3030 is turned to theviewpoint 3025 in the virtual space. In another state 300 c, thecross-section normal direction 3020 is not turned to the viewpoint 3025in the virtual space.

When the cross-section normal direction 3020 is not turned to theviewpoint 3025, as represented by 300 a, a cross-section normaldirection adjustment unit 1080 reverses the cross-section normaldirection 3020 and changes it so that the cross-section normal direction3030 is turned to the viewpoint 3025, like the cross-section normaldirection 3030 in 300 b. The cross-section normal direction adjustmentunit 1080 then updates the virtual information management unit 1060.

A cross-sectional operation direction decision unit 1100 decides across-sectional operation direction from the position and orientation ofthe viewpoint calculated by the viewpoint position and orientationcalculation unit 1050, and the cross sectional plane 2040 and normal2100 that are acquired by the cross section position directionacquisition unit 1070. When an input direction acquired by the operationdirection acquisition unit 1090 coincides with a cross-sectionaloperation direction decided by the cross-sectional operation directiondecision unit 1100, a cross-sectional position change unit 1110 movesthe cross-sectional plane 2040, by a predetermined distance, and updatesthe virtual information management unit 1060.

A virtual-image generation unit 1120 generates an image of the virtualspace viewed from a viewpoint having the position and orientationobtained by the viewpoint position and orientation calculation unit1050. An image output unit 1140 outputs, to the display unit 1030 of theHMD 1010, the image of the virtual space received from the virtual imagegeneration unit 1120. Accordingly, the image of the virtual spacecorresponding to the position and orientation of the viewpoint of theobserver is presented in front of the eyes of the observer who wears theHMD 1010 on his head.

FIG. 9 is a view showing an operation device of another form. FIG. 9shows an operation device of a form in which the observer grips theoperation unit 1160 having the display unit 1030, and operates thedirection input unit 1150 while viewing the virtual object 2090 andcross-sectional plane 2040 displayed on the display unit.

<Apparatus Operation>

FIG. 8 is a flowchart showing processing to be performed by theinformation processing apparatus 1040 according to the first embodiment.

In step S8010, the virtual information management unit 1060 reads outpieces of information of the virtual object 2090 and cross-sectionalplane 2040 from the information storage unit 1065.

In step S8030, the viewpoint position and orientation calculation unit1050 calculates the position and orientation (position and line-of-sightdirection) of a viewpoint in the sensor coordinate system 2060. Notethat the relative positions and orientations of the head position andorientation measurement unit 1020 and the observer's eye serving as theviewpoint are known in advance from the shape of the HMD 1010 and can beeasily calculated.

FIG. 4 is a view for explaining the operation in step S8030. Since therelative positions and orientations of the virtual space coordinatesystem 2080 and sensor coordinate system 2060 are known in advance, theposition and orientation of a viewpoint in the sensor coordinate system2060 is converted into a coordinate value in the virtual spacecoordinate system. Then, as shown in FIG. 4, a viewpoint position 4010,and three axes perpendicular to each other at this point, that is, a Cxaxis 4020 perpendicular to the visual axis in the horizontal direction,a Cy axis 4030 perpendicular to the visual axis in the verticaldirection, and a Cz axis 4040 along the visual axis are obtained.

If it is determined in step S8040 that the operation directionacquisition unit 1090 has received an input from the direction inputunit 1150, the process advances to step S8050. If it is determined thatthe operation direction acquisition unit 1090 has not received an input,the process advances to step S8100.

In step S8050, the cross-sectional position direction acquisition unit1070 acquires the position and orientation of a currently setcross-sectional plane from the virtual information management unit 1060,and calculates the normal vector of the cross-sectional plane. Thenormal vector can be obtained by calculating two vectors from threearbitrary successive vertices of the cross-sectional plane, and derivingthe outer product of the two vectors. The normal vector of thecross-sectional plane may be saved in advance in the information storageunit 1065.

In step S8060, as represented by 300 a in FIG. 3, the cross-sectionnormal direction adjustment unit 1080 obtains a viewpoint direction unitvector 3010 from a vector having, as a start point, one arbitrary point3015 on the cross section and, as an end point, the viewpoint 3025calculated by the viewpoint position and orientation calculation unit1050. Further, the cross-section normal direction adjustment unit 1080obtains the normal direction unit vector 3020 of the cross-sectionalplane calculated by the cross section position direction acquisitionunit 1070. Then, the cross-section normal direction adjustment unit 1080calculates the scalar product of the viewpoint direction unit vector3010 and the normal direction unit vector 3020 of the cross-sectionalplane.

A negative scalar product represents that the cross-section normaldirection 3020 serving as the cutting direction of the cross section ofthe virtual object is not turned to the line-of-sight direction. Thecross-section normal direction 3020 is therefore reversed so that theobserver can observe the cross section from the viewpoint position. Morespecifically, the cutting direction of the cross section of the virtualobject is changed so that it is turned to the viewpoint 3025, like thecross-section normal direction 3030 in 300 b. When the scalar product is0 or positive, the cross section is turned to the line-of-sightdirection, and nothing is done.

When the cutting direction of the cross section of the virtual object isnot turned to the line-of-sight direction, as in 300 c, even if thecross-section normal direction 3020 is reversed and turned to theviewpoint 3025, the line-of-sight direction is not turned to the cuttingdirection of the cross section. For this reason, neither the virtualobject nor the cross section of the virtual object is displayed on theHMD 1010. However, the HMD 1010 is mounted on the head and the positionand orientation of the viewpoint always can be changed. Since theobserver is highly likely to take an orientation in which theline-of-sight direction 3040 is turned to the virtual object and thecross section, it is preferable to reverse in advance the cuttingdirection of the cross section even in the case as represented by 300 c.

In step S8080, the cross-sectional operation direction decision unit1100 calculates an angle α formed by a direction perpendicular to theline of sight of the observer which, when standing in a normal way, isan “up” direction with respect to the observer; and the normal directionof the cross section. At this time, the scalar product of theup-with-respect-to-the-line-of-sight direction unit vector 4030 obtainedfrom the position and orientation of the viewpoint calculated by theviewpoint position and orientation calculation unit 1050, and the crosssection normal unit vector 2100 calculated by the cross-sectionalposition direction acquisition unit 1070 is obtained, as shown in FIG.4. According to known mathematical relationships, the scalar product ofthe unit vectors is the cosine value of an angle formed by these twovectors, so the formed angle α can be obtained from the cosine value.

FIG. 5 is a view for explaining an operation with respect to a crosssection turned to the up direction in step S8080. When thecross-sectional plane is turned to the up direction 4030 with respect tothe line of sight, as represented by 500 a and 500 b, the direction ofthe cross section's normal unit vector 2100 is associated with an “up”input to the direction input unit 1150. Also, a direction opposite tothe cross section's normal unit vector 2100 is associated with a “down”(that is, a direction opposite to “up”) input to the direction inputunit 1150. When the angle α falls within a predetermined angle range(0°≦α<(90−A)°, where 0<A<90, for example, 0°≦α≦45° when A is 45°), it isdetermined that the cross-sectional plane is turned to the up directionof the line of sight.

FIG. 6 is a view for explaining an operation with respect to a crosssection turned to the down direction in step S8080. When thecross-sectional plane is turned to the down direction of the line ofsight, as represented by 600 a and 600 b, the direction of the crosssection's normal unit vector 2100 is associated with a “down” input tothe direction input unit 1150. Thus, a direction opposite to the crosssection's normal unit vector 2100 is associated with an “up” input tothe direction input unit 1150. When the angle α falls within apredetermined angle range ((90+A)°<α≦180°, for example, 135°≦α≦180° whenA is 45°), it is determined that the cross-sectional plane is turned tothe down direction with respect to the viewpoint.

FIG. 7 is a view for explaining an operation with respect to a crosssection turned to a lateral direction (or sideways direction withrespect to the direction of gaze of the observer) in step S8080. Whenthe cross-sectional plane is turned to the lateral direction withrespect to the line of sight, as represented by 700 a to 700 d, a formedangle β is obtained from the scalar (or inner) product of aline-of-sight direction unit vector 7010 and the cross section's normalunit vector 2100. When the angle β falls within a predetermined anglerange ((90−B)°≦β<(90+B)°, where 0<B<90, for example, 45°<β<135° when Bis 45°), it is determined that the cross-sectional plane is turned in alateral direction with respect to the line of sight. The line-of-sightdirection 7010 is a direction (−Cz) opposite to the visual axisdirection vector 4040 (Cz) of the viewpoint in FIG. 4 obtained from theposition and orientation of the viewpoint calculated by the viewpointposition and orientation calculation unit 1050.

When the cross-section normal direction and the line-of-sight directionare turned to the same direction, as represented by 700 a, the directionof the cross section's normal unit vector 2100 is associated with a“forward” input to the direction input unit 1150. Then, a directionopposite to the cross section normal unit vector 2100 is associated witha “backward” input to the direction input unit 1150. When the angle βfalls within a predetermined angle range (0°≦β<(90−B)°, where 0<B<90,for example, 0°≦β<45° when B is 45°), it is determined that thecross-sectional plane and the line-of-sight direction are in the samedirection.

When the cross-section normal direction and the line-of-sight directionare turned to opposite directions (i.e. when they face each other), asrepresented by 700 b, the direction of the cross section's normal unitvector 2100 is associated with a “backward” input to the direction inputunit 1150. In addition, a direction opposite to the cross section normalunit vector 2100 is associated with a “forward” input to the directioninput unit 1150. When the angle β falls within a predetermined anglerange ((90+B)°<β≦180°, for example, 135°<β≦180° when B is 45°), it isdetermined that the cross-sectional plane and the line-of-sightdirection are in opposite directions.

When the cross-sectional plane is turned to the left or right directionwith respect to the line-of-sight direction when viewed from theobserver, as represented by 700 c or 700 d, the outer product of the xand y components of the line-of-sight direction unit vector 7010 and thex and y components of the cross section normal unit vector 2100 iscalculated. In this case, when the angle β falls within a predeterminedangle range (for example, 45°≦β≦135°), it is determined that thecross-sectional plane is turned to the “left” or “right” direction withrespect to the line-of-sight direction when viewed from the observer. Ifthe sign of the outer (or tensor) product is negative, it can bedetermined that the cross section is turned to the “right” with respectto the line-of-sight direction. If the sign is positive, it can bedetermined that the cross section is turned to the “left” with respectto the line-of-sight direction.

When the cross section is turned to the right with respect to theline-of-sight direction, the direction of the cross section's normalunit vector 2100 is associated with a “right” input to the directioninput unit 1150. In addition, a direction opposite to the crosssection's normal unit vector 2100 is associated with a “left” input tothe direction input unit 1150. In contrast, when the cross section isturned to the left direction with respect to the line-of-sightdirection, the direction of the cross section normal unit vector 2100 isassociated with a “left” input to the direction input unit 1150. Also, adirection opposite to the cross section normal unit vector 2100 isassociated with a “right” input to the direction input unit 1150.

In step S8090, if the input direction acquired by the operationdirection acquisition unit 1090 coincides with the cross-sectionaloperation direction decided by the cross-sectional operation directiondecision unit 1100, the cross-sectional position change unit 1110 movesthe cross-sectional plane 2040, by a predetermined distance, and updatesinformation managed in the virtual information management unit 1060.

In step S8100, the virtual image generation unit 1120 generates an imageof the virtual space viewed from the position and orientation of theviewpoint obtained by the viewpoint position and orientation calculationunit 1050. In step S8110, the image output unit 1140 outputs, to thedisplay unit 1030 of the HMD 1010, the image of the virtual spacereceived from the virtual image generation unit 1120.

If it is determined in step S8120 that an end request has been receivedfrom the observer, the process ends. If it is determined that no endrequest has been received, the process returns to step S8030 and isrepeated.

As described above, according to the first embodiment, the cross-sectionnormal direction is controlled based on the relationship between theviewpoint position of an observer (operator) and a cross-section normaldirection to which the cross section of a virtual object is turned. Inaddition, it is controlled to change the association between an input tothe direction input unit and the operation direction, based on therelationship between the line-of-sight direction of the observer(operator) and the cross-section normal direction to which the crosssection of the virtual object is turned. Hence, the first embodiment canimplement a natural cross-sectional operation with respect to athree-dimensional virtual object arranged in a virtual space.

Second Embodiment

The second embodiment will describe a form in which a head-mounteddisplay includes two display units as shown in FIG. 10. Morespecifically, an HMD 10010 includes two display units 10030 for theright and left eyes so as to provide a three-dimensional (3D) display toan observer.

<Apparatus Arrangement>

FIG. 10 is a block diagram showing the overall arrangement of a systemaccording to the second embodiment. FIG. 11 is a view showing the outerappearance of the head-mounted display (HMD) according to the secondembodiment. In FIGS. 10 and 11, the same reference numerals as those inFIGS. 1 and 2 denote the same parts, and a description thereof will notbe repeated.

The head-mounted display (HMD) 10010 includes the display units 10030for presenting images to the respective right and left eyes of anobserver who wears the HMD 10010 on his head, as represented by the view1100 a. The observer who wears the HMD 10010 on his head observes theleft-eye display unit 10030 (left side) with his left eye, and theright-eye display unit 10030 (right side) with his right eye. Thedisplay units 10030 display images of a virtual space sent from aninformation processing apparatus 10040. As a result, thethree-dimensional image of the virtual space corresponding to theposition and orientation of the left and right viewpoints of theobserver is presented in front of the eyes of the observer who wears theHMD 10010 on his head. Note that the HMD 10010 may include three or moredisplay units, and adopt an arrangement that presents virtual spaceimages corresponding to the positions and orientations of the viewpointsof the respective display units.

A viewpoint position and orientation calculation unit 10050 of theinformation processing apparatus 10040 calculates the positions andorientations (e.g. positions and line-of-sight directions) of the rightand left eyes in a virtual space coordinate system 2080 of the image ofa virtual space. Note that the relative positions and orientations of ahead position and orientation measurement unit 1020 and the observer'seye serving as the viewpoint are known in advance from the shape of theHMD 10010 and can be easily calculated. Note that it may be configuredto calculate one position and orientation that typifies two positionsand orientations, and update a virtual information management unit 1060.

A virtual image generation unit 10020 generates images of the virtualspace corresponding to the positions and orientations of the left andright viewpoints of the display units 10030. An image output unit 10140outputs the images corresponding to the left and right display units10030.

<Apparatus Operation>

FIG. 12 is a flowchart showing processing to be performed by theinformation processing apparatus 10040 according to the secondembodiment. In the flowchart shown in FIG. 12, the same referencenumerals as those in FIG. 8 denote the same processes, and a descriptionthereof will not be repeated.

In step S12030, the viewpoint position and orientation calculation unit10050 (see FIG. 10) calculates the typical position and orientation(position and line-of-sight direction) of a viewpoint. As in step S8030,a left viewpoint position 11021 of the display unit 10030, and threeaxes perpendicular to each other at this point, that is, a Cx axis 11031perpendicular to the visual axis in the horizontal direction, a Cy axis11041 perpendicular to the visual axis in the vertical direction, and aCz axis 11051 along the visual axis are obtained, as represented by 1100b of FIG. 11. Further, a right viewpoint position 11022 of the displayunit 10030, and three axes perpendicular to each other at this point,that is, a Cx axis 11032 perpendicular to the visual axis in thehorizontal direction, a Cy axis 11042 perpendicular to the visual axisin the vertical direction, and a Cz axis 11052 along the visual axis areobtained. Then, one typical position and orientation is calculated fromthe positions and orientations of the respective left and rightviewpoints.

First, the middle point of a straight line connecting the positions11021 and 11022 is obtained and set as a typical position 11020. The sumof the vectors of the Cy axes 11041 and 11042 is calculated to obtain aunit vector and set it as a typical orientation Cy axis 11140. Then, thesum of the vectors of the Cz axes 11051 and 11052 is calculated toobtain a unit vector and set it as a typical orientation Cz axis 11150.Further, the outer product of the Cy axis 11140 and Cz axis 11150 iscalculated to obtain a typical orientation Cx axis 11130 perpendicularto the Cy axis 11140 and the Cz axis 11150.

In the HMD 10010 including three or more display units, the barycenterof the viewpoint position of each display unit is set as the typicalposition. The sums of the vectors for the respective typical orientationaxes Cy and Cz are calculated to obtain unit vectors.

In step S12100, the virtual image generation unit 10020 generates imagesof the virtual space viewed from the positions and orientations of theleft and right viewpoints of the display units 10030. In step S12110,the image output unit 10140 outputs, to the left and right display units10030 of the HMD 10010, the images of the virtual space at the left andright viewpoints of the display units 10030 that have been received fromthe virtual image generation unit 10020.

As described above, according to the second embodiment, athree-dimensional virtual object arranged in a virtual space can bepresented to an observer (operator) by a three-dimensional (3D) display.As in the first embodiment, the second embodiment can implement anatural cross-sectional operation with respect to the three-dimensionalvirtual object arranged in the virtual space.

Third Embodiment

The third embodiment will describe a form in which a head-mounteddisplay further includes two image capturing units arranged at thepositions of the right and left eyes as shown in FIGS. 13 and 14. Morespecifically, an HMD 13010 includes image capturing units 13020 so as toprovide an observer with a mixed reality (MR) display in which the imageof a three-dimensional virtual object arranged in a virtual space issuperimposed and displayed on the scene of a physical space.

<Apparatus Arrangement>

FIG. 13 is a block diagram showing the overall arrangement of a systemaccording to the third embodiment. FIG. 14 is a view showing the outerappearance of the system according to the third embodiment. In FIGS. 13and 14, the same reference numerals as those in FIGS. 2, 10, and 11denote the same parts, and a description thereof will not be repeated.

The head-mounted display (HMD) 13010 is configured by further adding thetwo image capturing units 13020 to the HMD 10010. The image capturingunits 13020 capture moving images of the physical space, andsequentially input the captured images (images of the physical space) toa captured image acquisition unit 13050 of an information processingapparatus 13040. The two image capturing units 13020 are arranged forthe right and left eyes of an observer who wears the HMD 13010 on hishead. The image of the physical space captured by the right-eye imagecapturing unit 13020 is displayed on a right-eye display unit 10030, andthe image of the physical space captured by the left-eye image capturingunit 13020 is displayed on a left-eye display unit 10030.

The captured image acquisition unit 13050 sends the acquired images ofthe physical space to an image composition unit 13030. The imagecomposition unit 13030 generates composite images by superimposingimages of a virtual space generated by a virtual image generation unit10020, and the images of the physical space acquired from the capturedimage acquisition unit 13050. At this time, two composite images arecreated for the right and left eyes. The image composition unit 13030outputs the generated composite images to an image output unit 10140,and the image output unit 10140 outputs the composite images to thedisplay units 10030.

<Apparatus Operation>

FIG. 15 is a flowchart showing processing to be performed by theinformation processing apparatus 13040 according to the thirdembodiment. In the flowchart shown in FIG. 15, the same referencenumerals as those in FIG. 12 denote the same processes, and adescription thereof will not be repeated.

In step S15010, the captured image acquisition unit 13050 sends acquiredimages of the physical space to the image composition unit 13030. Instep S15020, the image composition unit 13030 generates composite imagesof images of the virtual space generated by the virtual image generationunit 10020, and the images of the physical space acquired from thecaptured image acquisition unit 13050. The image composition unit 13030outputs the generated composite images to the image output unit 10140,and the image output unit 10140 outputs the composite images to thedisplay units 10030.

As described above, the third embodiment can provide an observer(operator) with a mixed reality (MR) display in which the image of athree-dimensional virtual object arranged in a virtual space issuperimposed and displayed on the scene of a physical space. As in thefirst embodiment, the third embodiment can implement a naturalcross-sectional operation with respect to the three-dimensional virtualobject arranged in the virtual space.

Fourth Embodiment

The fourth embodiment will describe a form in which an operation unit(indicator) is used to perform an operation based on a change of theposition and orientation of the operation unit itself. In particular,the operation unit includes a magnetic sensor similar to a head positionand orientation measurement unit 1020 of the first embodiment, and isconfigured to be able to measure the position and orientation of theoperation unit.

<Apparatus Arrangement>

FIG. 16 is a block diagram showing the overall arrangement of a systemaccording to the fourth embodiment. FIG. 17 is a view showing the outerappearance of the system according to the fourth embodiment. In FIGS. 16and 17, the same reference numerals as those in FIGS. 1 and 2 denote thesame parts, and a description thereof will not be repeated.

Similarly to that described above, an operation unit 16160 includes anoperation unit position and orientation measurement unit 16020 that is amagnetic sensor similar to the head position and orientation measurementunit 1020. The operation unit position and orientation measurement unit16020 measures a change of a magnetic field corresponding to theposition and orientation of the operation unit position and orientationmeasurement unit 16020, and sends the measurement result to a sensorcontroller 2070. The sensor controller 2070 generates, from themeasurement result, a signal value representing the position andorientation of the operation unit position and orientation measurementunit 16020 in the sensor coordinate system, and sends the signal valueto an information processing apparatus 16040.

The operation unit 16160 also includes an event input unit 16150 that isoperated by a user in order to generate an event. For example, the eventinput unit 16150 is a button that can be pressed. When the user wants togenerate an event, he presses this button. If this button is pressed,the operation unit 16160 generates an event and outputs it to theinformation processing apparatus 16040.

<Apparatus Operation>

FIG. 18 is a flowchart showing processing to be performed by theinformation processing apparatus 16040 according to the fourthembodiment. In the flowchart shown in FIG. 18, the same referencenumerals as those in FIG. 8 denote the same processes, and a descriptionthereof will not be repeated.

In step S18040, an operation direction acquisition unit 16090 calculatesthe position and orientation of the operation unit 16160 in a virtualspace coordinate system 2080, similarly to the viewpoint position andorientation calculation unit 1050 in step S8030.

FIG. 19 is a view for explaining the operation in step S18040. First, asrepresented by 1900 a, a point 19030 is calculated, at which a straightline connecting a position 19010 of the operation unit 16160 and aviewpoint 19020 crosses a projection plane 19035 on which a virtualobject viewed from a viewpoint in a virtual space is projected.

Then, as represented by 1900 b, the vector of a straight line having, asa start point, a previous position 19040 of the operation unit 16160 onthe projection plane and, as an end point, a current position 19050 onthe projection plane is calculated to determine an up, down, left, orright input direction on the projection plane. When the previousposition on the projection plane and the current position on theprojection plane are completely the same, it is preferable not to startcalculation until the current position on the projection plane changesto a different position. Since the vector of the straight line is atwo-dimensional vector on the projection plane, it suffices to determinethe up, down, left, or right input direction based on the absolutevalues of the X and Y components of the vector. That is, a direction inwhich the absolute value increases is determined as an input direction.

Note that various other methods are conceivable as the method fordetermining an operation direction by the operation directionacquisition unit. For example, a vector obtained by projecting thecurrent indication direction of the operation unit 16160 on theprojection plane 19035 may be directly used in determination as the up,down, left, or right input direction.

Alternatively, specific keys of a keyboard may be assigned to up, down,left, and right directions and used in determination of the up, down,left, or right input direction. The moving direction of a mouse and aclick event may also be used in determination of the up, down, left, orright input direction. An audio input may also be used in determinationof the up, down, left, or right input direction. Further, the motion ofthe hand may be captured, and a gesture determined from the successivehand images may be used in determination of the up, down, left, or rightinput direction.

As described above, according to the fourth embodiment, an observer(operator) can operate a three-dimensional virtual object arranged in avirtual space by using the operation unit (indicator) that performs anoperation based on a change of the position and orientation of theoperation unit itself. As in the first embodiment, the fourth embodimentcan implement a natural cross-sectional operation with respect to thethree-dimensional virtual object arranged in the virtual space.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-006217, filed Jan. 16, 2014, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An information-processing apparatus comprising:an acquisition unit configured to acquire a line-of-sight direction ofan observer who observes a first three-dimensional virtual object havinga first cross section, arranged in a virtual space and having at leastone cross section; an image generation unit configured to generate,based on the line-of-sight direction of the observer and a first normalvector of the first cross section, a second three-dimensional virtualobject having a second cross section, wherein a second normal vectorwhich is a normal vector of the second cross section is different fromthe first normal vector; an output unit configured to output an image ofthe second three-dimensional virtual object generated by said imagegeneration unit; and an operation unit configured to accept anindication from the observer to move the second cross section, whereinsaid image generation unit is operable to generate the secondthree-dimensional virtual object in a state in which the second crosssection is moved by a predetermined distance in a direction of thenormal vector of the second cross section based on an operation by saidoperation unit, said operation unit is configured to accept anindication of a first direction, an indication of a second directionopposite to the first direction, an indication of a third directionperpendicular to the first direction, and an indication of a fourthdirection opposite to the third direction, and in a case in which anangle β formed by the line-of-sight direction and a normal direction ofthe second cross section falls within a first angle range, said imagegeneration unit is operable to generate the second three-dimensionalvirtual object in the state in which the second cross section is movedin a first moving direction based on the indication of the firstdirection in said operation unit and to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in a second moving direction opposite to the firstmoving direction based on the indication of the second direction in saidoperation unit, in a case in which the angle β falls within a secondangle range different from the first angle range, said image generationunit is operable to generate the second three-dimensional virtual objectin the state in which the second cross section is moved in the secondmoving direction based on the indication of the first direction in saidoperation unit and to generate the second three-dimensional virtualobject in the state in which the second cross section is moved in thefirst moving direction based on the indication of the second directionin said operation unit, in a case in which the angle β falls within athird angle range different from the first angle range and the secondangle range, and a sign of a value of an outer product of a vector ofthe line-of-sight direction and the normal vector of the secondcross-section is positive, said image generation unit is operable togenerate the second three-dimensional virtual object in the state inwhich the second cross section is moved in a third moving directionbased on the indication of the third direction in said operation unitand to generate the second three-dimensional virtual object in the statein which the second cross section is moved in a fourth moving directionopposite to the third moving direction based on the indication of thefourth direction in said operation unit, and in a case in which theangle β falls within the third angle range and the sign of the value ofthe outer product of the vector of the line-of-sight direction and thenormal vector of the second cross-section is negative, said imagegeneration unit is operable to generate the second three-dimensionalvirtual object in the state in which the second cross section is movedin the fourth moving direction based on the indication of the thirddirection in said operation unit and to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in the third moving direction based on the indicationof the fourth direction in said operation unit.
 2. The apparatusaccording to claim 1, wherein the second normal vector of the secondcross section is a vector obtained by reversing the first normal vectorof the first cross section.
 3. The apparatus according to claim 1,wherein the first angle range is 0°≦β<(90−B)°, where 0<B<90, the secondangle range is (90+B)°<β≦180°, and the third angle range is(90−B)°≦β<(90+B)°q.
 4. The apparatus according to claim 1, wherein saidoperation unit is configured as a four-way selector key capable ofindicating two axial directions.
 5. The apparatus according to claim 1,wherein said output unit is configured to output the image of the secondthree-dimensional virtual object on a display device configured as ahead-mounted display (HMD).
 6. The apparatus according to claim 5,wherein the head-mounted display (HMD) includes two display unitsconfigured to present images to respective right and left eyes of theobserver, and said image generation unit is operable to generate twoimages corresponding to the two respective display units.
 7. Theapparatus according to claim 6, the head-mounted display (HMD) furtherincludes two image-capturing units configured to capture a scene of aphysical space from positions of the respective right and left eyes ofthe observer, and said image generation unit is operable to generate animage in which the second three-dimensional virtual object in thevirtual space is superimposed and displayed on two captured imagesobtained by the two image-capturing units as the two imagescorresponding to the two respective display units.
 8. The apparatusaccording to claim 1, wherein said image generation unit is operable todetermine the second normal vector based on a scalar product of thefirst normal vector and a vector which is determined on the basis of theline-of-sight direction of the observer and a position of the firstcross section, and to generate the second three-dimensional virtualobject having the second cross section.
 9. An information processingmethod to be executed by an image-processing apparatus, comprising:acquiring a line-of-sight direction of an observer who observes a firstthree-dimensional virtual object having a first cross section, arrangedin a virtual space and having at least one cross section; generating,based on the line-of-sight direction of the observer and a first normalvector of the first cross section, a second three-dimensional virtualobject having a second cross section, wherein a second normal vectorwhich is a normal vector of the second cross section is different fromthe first normal vector; outputting an image of the generated secondthree-dimensional virtual object; and accepting an indication from theobserver at an operation unit to move the second cross section, whereinthe image-processing apparatus is operable to generate the secondthree-dimensional virtual object in a state in which the second crosssection is moved by a predetermined distance in a direction of thenormal vector of the second cross section based on an operation by theimage-processing apparatus, the operation unit is configured to acceptan indication of a first direction, an indication of a second directionopposite to the first direction, an indication of a third directionperpendicular to the first direction, and an indication of a fourthdirection opposite to the third direction, and in a case in which anangle β formed by the line-of-sight direction and a normal direction ofthe second cross section falls within a first angle range, theimage-processing apparatus is operable to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in a first moving direction based on the indication ofthe first direction in the operation unit and to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in a second moving direction opposite to the firstmoving direction based on the indication of the second direction in theoperation unit, in a case in which the angle β falls within a secondangle range different from the first angle range, the image-processingapparatus is operable to generate the second three-dimensional virtualobject in the state in which the second cross section is moved in thesecond moving direction based on the indication of the first directionin the operation unit and to generate the second three-dimensionalvirtual object in the state in which the second cross section is movedin the first moving direction based on the indication of the seconddirection in the operation unit, in a case in which the angle β fallswithin a third angle range different from the first angle range and thesecond angle range, and a sign of a value of an outer product of avector of the line-of-sight direction and the normal vector of thesecond cross-section is positive, the image-processing apparatus isoperable to generate the second three-dimensional virtual object in thestate in which the second cross section is moved in a third movingdirection based on the indication of the third direction in theoperation unit and to generate the second three-dimensional virtualobject in the state in which the second cross section is moved in afourth moving direction opposite to the third moving direction based onthe indication of the fourth direction in the operation unit, and in acase in which the angle β falls within the third angle range and thesign of the value of the outer product of the vector of theline-of-sight direction and the normal vector of the secondcross-section is negative, the image-processing apparatus is operable togenerate the second three-dimensional virtual object in the state inwhich the second cross section is moved in the fourth moving directionbased on the indication of the third direction in the operation unit andto generate the second three-dimensional virtual object in the state inwhich the second cross section is moved in the third moving directionbased on the indication of the fourth direction in the operation unit.10. A non-transitory computer-readable storage medium storing a programfor causing a computer to execute an information processing methodaccording to claim 9 comprising the steps of: acquiring a line-of-sightdirection of an observer who observes a first three-dimensional virtualobject having a first cross section, arranged in a virtual space andhaving at least one cross section; generating, based on theline-of-sight direction of the observer and a first normal vector of thefirst cross section, a second three-dimensional virtual object having asecond cross section, wherein a second normal vector which is a normalvector of the second cross section is different from the first normalvector; outputting an image of the generated second three-dimensionalvirtual object; and accepting an indication from the observer at anoperation unit to move the second cross section, wherein the programcauses the computer to generate the second three-dimensional virtualobject in a state in which the second cross section is moved by apredetermined distance in a direction of the normal vector of the secondcross section based on an operation by the image-processing apparatus,the operation unit is configured to accept an indication of a firstdirection, an indication of a second direction opposite to the firstdirection, an indication of a third direction perpendicular to the firstdirection, and an indication of a fourth direction opposite to the thirddirection, and in a case in which an angle β formed by the line-of-sightdirection and a normal direction of the second cross section fallswithin a first angle range, the program causes the computer to generatethe second three-dimensional virtual object in the state in which thesecond cross section is moved in a first moving direction based on theindication of the first direction in the operation unit and to generatethe second three-dimensional virtual object in the state in which thesecond cross section is moved in a second moving direction opposite tothe first moving direction based on the indication of the seconddirection in the operation unit, in a case in which the angle β fallswithin a second angle range different from the first angle range, theprogram causes the computer to generate the second three-dimensionalvirtual object in the state in which the second cross section is movedin the second moving direction based on the indication of the firstdirection in the operation unit and to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in the first moving direction based on the indicationof the second direction in the operation unit, in a case in which theangle β falls within a third angle range different from the first anglerange and the second angle range, and a sign of a value of an outerproduct of a vector of the line-of-sight direction and the normal vectorof the second cross-section is positive, the program causes the computerto generate the second three-dimensional virtual object in the state inwhich the second cross section is moved in a third moving directionbased on the indication of the third direction in the operation unit andto generate the second three-dimensional virtual object in the state inwhich the second cross section is moved in a fourth moving directionopposite to the third moving direction based on the indication of thefourth direction in the operation unit, and in a case in which the angleβ falls within the third angle range and the sign of the value of theouter product of the vector of the line-of-sight direction and thenormal vector of the second cross-section is negative, the programcauses the computer to generate the second three-dimensional virtualobject in the state in which the second cross section is moved in thefourth moving direction based on the indication of the third directionin the operation unit and to generate the second three-dimensionalvirtual object in the state in which the second cross section is movedin the third moving direction based on the indication of the fourthdirection in the operation unit.
 11. An information-processing apparatuscomprising: an acquisition unit configured to acquire a line-of-sightdirection of an observer who observes a first three-dimensional virtualobject having a first cross section, arranged in a virtual space andhaving at least one cross section; an image generation unit configuredto generate, based on the line-of-sight direction of the observer and afirst normal vector of the first cross section, a secondthree-dimensional virtual object having a second cross section, whereina second normal vector which is a normal vector of the second crosssection is different from the first normal vector; an output unitconfigured to output an image of the second three-dimensional virtualobject generated by said image generation unit; and an operation unitconfigured to accept an indication from the observer to move the secondcross section, wherein said image generation unit is operable togenerate the second three-dimensional virtual object in a state in whichthe second cross section is moved in a direction of the second normalvector based on an operation by said operation unit, wherein saidoperation unit is configured to accept, from the observer, at least anindication of a first direction and an indication of a second directionopposite to the first direction, and in a case in which an angle αformed between a direction of the second cross section and an upwarddirection relative to the observer falls within a first angle range,said image generation unit is operable to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in a first moving direction based on the indication ofthe first direction in said operation unit and to generate the secondthree-dimensional virtual object in the state in which the second crosssection is moved in a second moving direction opposite to the firstmoving direction based on the indication of the second direction in saidoperation unit, and in a case in which the angle α falls within a secondangle range different from the first angle range, said image generationunit is operable to generate the second three-dimensional virtual objectin the state in which the second cross section is moved in the secondmoving direction based on the indication of the first direction in saidoperation unit and to generate the second three-dimensional virtualobject in the state in which the second cross section is moved in thefirst moving direction based on the indication of the second directionin said operation unit, and wherein the first angle range is0°≦α<(90−A)°, where 0<A<90, and the second angle range is(90+A)°<α≦180°.